Solving the neutron transport equation by the Monte Carlo method is computationally expensive. The Monte Carlo method works by tracing a neutrons random walk, and doing this repeatedly for a large number of histories. Engineers have developed a correlated sampling method to avoid repeating expensive MC calculations for similar models.

The correlated sampling method logs key data from each history as it is traced, then retraces the history through each alternate model to calculate a correction factor. The correction factor accounts for the differences between the models by adjusting the history’s weight. Each history therefore collects information about the original model (from the trace) and information about each model in which it is retraced. (The trace and retrace data are correlated, so it is inappropriate to retrace a history through the model in which it was originally traced.)

The retrace step, per model, is generally much faster than the corresponding trace. The trace logs the nuclear data it retrieves, so the retrace can retrieve the data sequentially from the trace log instead of seeking it in RAM. Therefore using correlated sampling across multiple universes is much faster than performing an independent calculation in each. The retrace burden will be significant, however, if histories are retraced through many models.



\subsection{CUDA-Accelerated Retracing}

Monte Carlo radiation transport (i.e., tracing) is usually difficult to accelerate with GPUs. The branching statements, random memory lookups, and random tally incrementing are not efficient on GPU architectures. The retrace step, however, requires large amounts of addition, multiplication, and exponentiation with little branching and structured tally incrementing.

In problems in which many models are compared, the retrace burden is significant. For example, inverse problems analyzing the contents of unknown packages need to compare many similar models to solve for the gradient of a cost function. In nuclear reactor simulations, operators typically simulate interactions in thousands of models with varying temperatures and irradiation profiles. In these problems, GPU acceleration may significantly reduce time to solution.



\subsection{Retracing and MPI}

Correlated sampling, like the Monte Carlo algorithm on which it’s based, is embarrassingly parallel. As a processor traces a history, it can simultaneously retrace the history through all models. In that implementation, all models' nuclear data and tallies must be available to each process.

In high-fidelity reactor simulations, the tally matrix or model compositions are too large to fit on a single node. In that case, it is necessary to domain decompose the models among several nodes. Each processor traces histories through a common trace model and retraces the logs through the local models. It then exchanges trace logs with other processors and retraces the histories they traced into the local models. Each process traces with different random numbers.

For most problems, trace logs are too large to send after all traces are complete. The average size of the trace logs is proportional to the number of histories traced. Therefore logs must be exchanged and retraced in batches of an optimial size. Batches with too many histories will occupy an excessive memory footprint. Batches with too few histories will reduce performance due to communication latency and dependency barriers.
